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A single two-dimensinoal Dirac fermion state has been recently observed on the surface of topolog- 
ical insulator Bi 2 Te3 by angle-resolved photoemission spectroscopy (ARPES). We study the surface 
band structure using k • p theory and find an unconventional hexagonal warping term, which is the 
counterpart of cubic Dresselhaus spin-orbit coupling in rhombohedral structures. We show that 
this hexagonal warping term naturally explains the observed hexagonal snow-flake Fermi surface. 
The strength of hexagonal warping is characterized by a single parameter, which is extracted from 
the size of the Fermi surface. We predict a number of testable signatures of hexagonal warping in 
spectroscopy experiments on Bi2Te3. We also explore the possibility of a spin-density wave due to 
strong nesting of the Fermi surface. 
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Recently a new state of matter called topological in- 
sulators has been observed in a number of materials [1, 
HJ |3j [4j [5] . A topological insulator has a time-reversal- 
invariant band structure with nontrivial topological or- 
der, which gives rise to gapless surface states bound 
to the sample boundary [6j [7J [8]. The two-dimensional 
surface band has a unique Fermi surface that encloses 
an odd number of Dirac points in the surface Brillouin 
zone [6], which is prohibited in conventional materials by 
fermion doubling theorem [9 j. Soon after the theoretical 
prediction[10 , the semiconducting alloy Bi^Sbi-^ was 
found to be a topological insulator having a Dirac sur- 
face band as well as other electron and hole pockets pQ. 
Subsequently, a family of materials 1^2X3 (X=Se and 
Te) was found to be topological insulators with a single 
Dirac- fermion surface state [3j 0J |5j. The observation of 
an undoubled Dirac fermion is not only of great concep- 
tual interest but also paves the way for studying unusual 
electromagnetic properties [TQl [11] and realizing topolog- 
ical quantum computation [1 2 . Therefore surface states 
of 1^2X3 are being intensively studied in transport and 
spectroscopy experiments [I3j [14]. 

In this work, we study the electronic properties of sur- 
face states in Bi2Te3 using k • p theory. Our motivation 
is to understand the shape of Fermi surface observed in 
recent ARPES experiments [4[ [5], reproduced in Fig.l. 
By considering the crystal symmetry of Bi2Te3, we find 
an unconventional hexagonal warping term in the sur- 
face band structure, which is the counterpart of cubic 
Dresselhaus spin-orbit coupling in rhombohedral struc- 
tures. This hexagonal warping term naturally explains 
the snow-flake shape of the Fermi surface, and its magni- 
tude is extracted from the size of Fermi surface. We pre- 
dict that hexagonal warping of the Fermi surface should 
have important effects in several spectroscopy experi- 
ments. Finally, we observe that Fermi surface of Bi2Te3 
is nearly a hexagon with strong nesting for an appro- 
priate range of surface charge density. This motivates 
us to explore theoretically a possible spin-density wave 
(SDW) phase. We discuss various types of SDW order in 




FIG. 1: (i) snow-flake like Fermi surface of the surface states 
on 0.67% Sn-doped Bi 2 Te 3 observed in ARPES. (ii) a set 
of constant energy contours at different energies. From [4]. 
Reprinted with permission from AAAS. 



a Landau-Ginzburg theory. 

Bi2Te3 has a rhombohedral crystal structure with 
space group R3fh. In the presence of a [111] surface, 
the symmetry of the crystal is reduced to C% V1 which 
consists of a three-fold rotation C3 around the trigonal 
z axis and a mirror operation M : x — > — x where x 
is in YK direction. Two surface bands are observed to 
touch at the origin of the surface Brillouin zone T. The 
degeneracy is protected by time-reversal symmetry and 
the doublet IV^l) form a Kramers pair. We choose a 
natural basis for the doublet according to total angular 
momentum J = L + S = ±1/2 so that C3 is represented 
as e -mz7r / 3 . Since M 2 = —1 for spin 1/2 electron and 
MC^M -1 = C3" 1 , the mirror operation can be repre- 
sented as M = ia x by defining the phase of |^|,|) ap- 
propriately. The anti-unitary time reversal operation 
is represented by i<j y K (K is complex conjugation) and 
commutes with both M and C3. Here the pseudo-spin 
Gi is proportional to electron's spin: (s z ) cx (a z ) and 

The Kramers doublet is split away from Y by spin-orbit 
interaction. We study the surface band structure near Y 
using k-p theory. To lowest order in fc, the 2x2 effective 
Hamiltonian reads Hq = v(k x 0~ y ky 0~ % ), which describes 
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an isotropic 2D Dirac fermion. The form of Hq is strictly 
fixed by symmetry. In particular, the Fermi velocity v in 
x and y directions are equal because of the C3 symmetry. 
The Fermi surface of Hq at any Fermi energy is a circle. 
However, the Fermi surface observed in ARPES, repro- 
duced in Fig.l(i), is non-circular but snow-flake like: it 
has relatively sharp tips extending along six YM direc- 
tions and curves inward in between. Moreover, as shown 
in Fig. 1(h) (we refer the reader to the original work [4 
for better resolution), the shape of constant energy con- 
tour is energy-dependent, evolving from a snow-flake at 
E = 0.25eV to a hexagon and then to a circle near the 
Dirac point. Throughout this paper, energy is measured 
with respect to the Dirac point. 

The observed anisotropic Fermi surface can only be 
explained by higher order terms in the k • p Hamiltonian 
H(k) that break the emerging U(l) rotational symmetry 
of Ho. The form of H(k) is highly constrained by crystal 
and time reversal symmetry. Under the operation of C3 
and M, momentum and spin transform as follows: 

n . h. „±i2ir/?>i ^ _^ +i27r/3^ ^ , ^ 

C3 : k± — > e ; /c±, a± — > e 1 cr±, cr z — > a z 
M : k+ <-> cr x -> a x , a VyZ -> -cr^, (1) 

where fc± = fc^ ± zfc^ and <j± = <j x ± zcr^. if(fe) must be 
invariant under 0. In addition, time reversal symmetry 
gives the constraint 

H(k) = GHi-^G- 1 = a y H*(-k)a y . (2) 

We then find that H(k) must take the following form up 
to third order in k: 

H(k) = E (k) + v k (k x (Ty - k y a x ) + ^(k^ + k?_)cr z , (3) 

where E$(k) = /c 2 /(2m*) generates particle- hole asym- 
metry and the Dirac velocity = u(l + a/c 2 ) contains 
a second-order correction. The last term in (|3|, which 
we call H w , is most important. Unlike the other terms, 
H w is only invariant under three-fold rotation (as the 
Bi2Te3 crystal structure does) and therefore is solely re- 
sponsible for the hexagonal distortion of the otherwise 
circular Fermi surface. We note that H w (k) vanishes in 
mirror-symmetric direction TM, because a z is odd un- 
der mirror, and H w (k) reaches maximum along YK. The 
surface band dispersion of H(k) is 

E±(k) = Eo(k) ± ^v 2 k k 2 + \ 2 k 6 cos 2 (3(9). (4) 

Here E± denote the energy of upper and lower band, and 
is the azimuth angle of momentum k with respect to 
the x axis (YK). Although the Hamiltonian H is three- 
fold invariant, the band structure is six- fold symmetric 
under 6 — > 6 + 2tt/6 because of time reversal symmetry. 

The hexagonal warping term H w describes cubic spin- 
orbit coupling at the surface of rhombohedral crystal sys- 
tems, and to the best of our knowledge, it has not been 



reported before. It is instructive to compare H w with 
the well-studied trigonal warping in graphene[15 . Al- 
though graphene's band structure also has Dirac points 
and its k ■ p Hamiltonian Hx(k) has Cs v symmetry, the 
warping term in graphene is of a completely different 
form. This is because time reversal operation acts dif- 
ferently for spin 1/2 (6 = ia y K) and spinless fermions 
(O = K). In graphene time reversal symmetry takes the 
latter form, and together with inversion symmetry, leads 
to Hjc(k) = r x H^(k)r x (r z = ±1 denote two sublat- 
tice), as opposed to its partner Eq.Q in Bi2Te3. As a 
result, a different trigonal warping term (& 2 _r + + k 2 _ r_) 
is symmetry- allowed in graphene. 
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FIG. 2: Left: constant energy contour of H(k). k x and k y 
axis are in the unit of \/v/X. Right: constant energy contour 
at E — 1.2E* is superimposed on the Fermi surface of Bi2Te3. 

We now show that H w naturally explains the observed 
energy-depdent shape of Fermi surface in Bi 2 Te3. Using 
Q we plot a set of constant energy contours of H(k) for 
< E < 2E* in Fig.2, where E* = v/a and a = ^J\jv 
are the characteristic energy and length scale introduced 
by hexagonal warping. For simplicity, we have discarded 
and the quadratic correction to velocity, since they 
do not change the shape of Fermi surface significantly. 
By plotting k x and k y axis in the unit of ^/v/\, Fig.2 is 
obtained with no free parameter. As shown in the figure, 
Fermi surface starts to deviate considerably from a cir- 
cle and becomes more hexagon-like around E = 0.55E*. 
When E > E c = V7/6 3/4 E* w 0.69£*, the edge of 
the hexagon curves inward so that Fermi surface ceases 
to be purely convex. As E further increases, rounded 
tips starts to develop at the vertices of the hexagon, 
which eventually become sharper making the Fermi sur- 
face snow-flake like. The evolution of Fermi surface with 
respect to energy matches well with the ARPES result 
shown in Fig.l. Moreover, it follows from Q that the 
vertices of the hexagon — where the Fermi surface extends 
outmost — always lie along YM independent of the sign 
of A, in agreement with ARPES data. 

Comparing the set of Fermi surfaces in Fig. 2a with the 
real Fermi surface in 0.67% Sn-doped Bi2Te3 (Fig.l), we 
find the Fermi surface at Ep = 1.2E* is almost identical 
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to the one measured in ARPES, as shown by superimpos- 
ing the two in Fig. 2b. By fitting the theoretical value of 
Fermi momentum along TM (1.2/a) to the experimental 
one (O.llA -1 ), we find a = 10. 9 A. Using the measured 
Fermi velocity v = 2.55eV ■ A, we obtain the magnitude 
of the hexagonal warping term: A = 250eV • A 3 . From 
that we find E* = 0.23eV, and E F = 1.2E* = 0.28eV 
which agrees fairly well with the measured Fermi energy 
0.25eV (shown in Fig.l.ii). The quantitative agreement 
between theory and experiment suggests that the Hamil- 
tonian ([3| describes the surface band structure of Bi2Te3 
quite well in a wide energy window at least up to 0.2beV. 
As an independent check of the theory, we consider the 
non-linear correction to surface band dispersion near T. 
Q predicts that the leading order correction due to H w 
starts at fifth order in k and is angle-dependent: 

SE(k,0) = va 4 k 5 cos 2 (30) /2. (5) 

Since the surface band dispersions along TK and TM di- 
rections have been measured in ARPES [HE], Q can be 
tested by fitting to ErM(k) — Erx(k)^ which also gives 
an independent way of obtaining A. The two other pa- 
rameters m* and a in H(k) can also be extracted by a 
careful fitting to the band dispersion. 

From now on, we predict a variety of important effects 
of hexagonal warping in Bi2Te3. First, because H w cou- 
ples to a z , the spin polarization of surface states should 
have an out-of-plane component s z oc (cr z ). Since spin 
polarization along TM has been found to be almost 100% 
polarized in a very recent ARPES experiment [5 , we con- 
clude that the the doublet |^T,l) a ^ ^ are almost pure spin 
eigenstates, i.e., s z ~ which agrees with a theoret- 
ical band structure calculation [16 . s z is then calculated 
from (j|): s z = cos(3(9)/ ^ cos 2 (30) + l/(/ca) 4 . The out- 
of-plane spin polarization is momentum-dependent and 
can reach as high as 60% of the full polarization along 
TK for the Fermi surface in Fig.l. We hope this pattern 
of out-of-plane spin polarization can be tested in future 
spin-resolved ARPES. 

Second, hexagonal warping gives a novel mechanism 
for opening up an energy gap at the Dirac point. Con- 
sider an in-plane magnetic field Bn, which only couples 

to the spin H\ eeman = g\\B\\ • a. From ^ we find the 
Dirac point is shifted away from r to k* = g\\z x Bu/v. 
In addition to that, a mass term is generated at k*: 
Ma z = (#|i £?||) 3 sin(3(p)a z / E* 2 ((f is the angle between 
B\\ and TK), which opens up an energy gap. When the 
Fermi energy is tuned, e.g. by doping[4, 5 , to lie within 
the gap, the insulating state at the surface realizes quan- 
tum Hall effect without Landau levels [T7]. 

Third, hexagonal warping of Fermi surface has drastic 
effects on the Friedel oscillation of local density of states 
(LDOS) around a nonmagnetic point defect in STM. The 
LDOS oscillation at a fixed energy decays algebraically as 
a function of distance away from the defect. In a normal 




FIG. 3: Illustration of scattering processes due to a point de- 
fect that causes the oscillation of LDOS. In a given direction x 
along TM, the oscillation is dominated by scattering between 
stationary points marked by dot, where the Fermi velocity 
is parallel to x. (a) a convex constant energy contour has a 
single pair of stationary point at k and —k. (b) a non-convex 
constant energy contour has three pairs of stationary points. 
Intra-pair scatterings in (a) and (b) are forbidden by time 
reversal symmetry. But inter-pair scatterings in (b), for ex- 
ample, those between ki and fe, are allowed. Therefore, the 
LDOS oscillation at leading order is absent in (a) but exists 
in (b). 



2D metal, the leading order (l/x) decay of LDOS in a 
given direction x comes from scattering between states 
at "stationary points" on the Fermi surface, where the 
Fermi velocity is parallel to x[18 . For a convex con- 
stant energy contour below E c as shown in Fig. 3a, only 
a single pair of stationary points exist at k and — k. 
However, since the two states at k and —k here carry 
opposite spins, scattering between them is forbidden by 
time reversal symmetry — a fundamental property of sur- 
face states on a topological insulator. The LDOS os- 
cillation then vanishes at leading order. Now consider a 
non-convex constant energy contour above E c . As shown 
in Fig. 3b, multiple pairs of stationary points exist. Since 
inter-pair scattering is still allowed, the LDOS oscillation 
will be restored at leading order. Therefore according 
to the convexity of constant energy contour, two types 
of Friedel oscillation patterns should appear at different 
ranges of bias voltage. 

In the last part of this work, we explore the possibil- 
ity of a spin-density wave (SDW) phase on the surface 
of Bi2Te3. We note that the Fermi surface is nearly a 
hexagon for 0.55E* < E < 0.9E*. The almost flat pieces 
on the edges of the hexagon leads to strong nesting at 
wave-vectors Qi = 2kpe il i = 1, ...3, where kp^ is the 
Fermi momentum in three equivalent TK directions. A 
density-wave ordered phase may then exist at a finite 
interaction strength. Since the surface states at k and 
— k have opposite spins, a charge- density wave cannot 
connect them and is thus disfavored. We are therefore 
motivated to consider possible SDW phases. 

We now discuss the phase diagram of SDW in a 
Landau-Ginzburg theory based on general symmetry 
considerations. We define the order parameters of the 
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SDW as follows, 

k 

<l>i± = i ^Z{ c l+QM x e *) ' &Ck ) 

k 

(t>iz = i J2^k+Qi aZc k) ( 6 ) 
k 

where c\ = (c| fc , cj fc ) are electron creation operators. For 
each Qi, we have chosen a local frame for in-plane spin 
components labeled by || and _L which are parallel and 
perpendicular to Qi respectively. The order parameters 
thus defined transform nicely under the operations of ro- 
tation, mirror, time reversal and translation: 



C 3 : 






6 : 






M x : 




0^, 02 M 03 M . 


T d : 


0i/i 





We remark that because spin and momentum are locked 
by spin-orbit coupling, there is no SU(2) symmetry for 
spin alone. Thanks to the appropriate choice of order pa- 
rameters, symmetry operations ^ only act in the space 
of ordering wave vectors labelled by i index. 

The Landau free energy F must be invariant under 
these symmetry operations. Only terms with even powers 
of can exist because of time reversal symmetry. At 
second order, we have 

1 3 

F 2 = ^X^v^^iv, (8) 

i=l 

where the susceptibility matrix x^v is real and symmet- 
ric because of mirror symmetry. \\iv is positive def- 
inite in the normal state. When the temperature is 
lowered below T c , one of the eigenvalues of X\iv first 
becomes negative, and the surface undergoes a transi- 
tion to a SDW. The spin configuration is then deter- 
mined by the corresponding eigenvector v^. For ex- 
ample, for a stripe SDW along x direction, S(x, y) = 
(i>ll cos(Qx), v±_ sin(Qx), v z sm(Qx)) with an appropriate 
choice of origin. 

The free energy ([8| to second order has an emerging 
U(3) symmetry c/) ifJL — > Uij(j)j^. So single- and multiple-Q 
SDWs are degenerate. We now show that higher order 
terms in F break the U(3) symmetry and picks out a 
particular spatial ordering pattern. For that purpose, it 
is convenient to write 0^ = ^v^, J2 i |£i| 2 = 1 and use & 
as a new set of order parameters, which also transforms 
according to (|7|). At fourth order, we find an anisotropy 
term F4 = v>Y^=i |£i| 4 - The sign of u determines the 
relative weight of fa in the ordered phase. For u < 0, 
only one of say ^1, is nonzero. The resulting SDW 



forms a one-dimensional stripe, which breaks Cs but is 
invariant under mirror symmetry. For u > 0, |£i| = 
I £2 1 = I £3 1 m the ordered phase, so that SDW forms a two- 
dimensional lattice. Each individual phase of ^ depends 
on the choice of origin. Only the global phase of £i£2^3 
is gauge invariant and is fixed by the sixth-order term of 
the form C(0i0 2 3 ) 2 + C*(0i0205) 2 in F - 

Note added: During the final stage of this work, we 
learned that Alpichshev et al.[19 imaged with STM the 
standing wave of surface states on Bi2Te3 near a line 
defect (instead of a point defect considered in this work) . 
The LDOS oscillation was found to exist in the energy 
range with snow-flake like constant energy contour, but 
strongly suppressed in the range with circular constant 
energy contour. This supports our explanation of the 
correlation between LDOS oscillation and convexity of 
constant energy contour. 
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